EHR Data QC Pipeline

This script generates tables and figures for EHR data quality control (QC). It processes NLP and codified datasets, ensuring data consistency and reliability for analysis.

Requirements

You should have two datasets (NLP & codified) that include at least:

• patient_num: Character variable for patient ID
• feature_id: Character variable for feature (code) ID
• start_date: Date of each feature_id

Additionally, you need a data dictionary containing:

• feature_id: Character variable for feature (code) ID
• description: Text description of feature_id

Setup

Define the following in Module 1:

• Directory paths to dictionary, NLP, and codified data
• target_code: PheCode of primary interest in your study
• target_cui: Corresponding CUI of primary interest in your study

Note that if you are not running this code on O2, you will need to download codified and NLP features from the ONCE webapp and manually specify directory paths for these dictionaries.

Module 0: Sample QC Data

This optional module samples 1,000 unique patients from the intersection of the NLP and codified datasets to speed up QC processing.

Outputs

1. Sampled NLP dataset (to be used in Module 1)
2. Sampled codified dataset (to be used in Module 1)

Module 1: Data and Dictionary Import & Preparation

This module imports and prepares the NLP and codified datasets for analysis. It also imports three data dictionaries - an institution-specific data dictionary with codified feature descriptions (user-defined) and two ONCE dictionaries for selecting similar features to the target PheCode and CUI (automatically uploaded from O2).

Outputs

1. Cleaned NLP and codified datasets
2. Filtered data dictionary with target and common code feature descriptions
3. ONCE dictionaries with selected codified and NLP features

Module 2: Patient, Code, and Follow-up Summaries

This module summarizes the NLP and codified datasets, including patient counts, prevalence of the target PheCode and CUI, and duration of patient follow-up. Patient counts are summarized annually (line plots) and overall (bar plots).

Outputs

1. Patient counts over time
2. Table of follow-up duration statistics

Total Sample Size

Dataset	Number of Patients
NLP	1000
Codified	1000

Module 3: Common PheCode Trends (Parent-Child Correlation)

This module examines hierarchical relationships between PheCodes, tracking trends over time for parent and child PheCodes.

Outputs

1. Plot of rates over time for 3 common PheCodes
2. Plot of patient counts over time for 3 common PheCodes

Notes: (1) Rates are calculated as the number of patients with the target PheCode per calendar year divided by the total number of patients with any code in the same year.

Module 4: Target PheCode-CUI Trends

This module analyzes the relationship between a target PheCode and CUI, including trends in annual rates, patient counts, and intra-patient correlation.

Outputs

1. Plot of rates over time for target PheCode-CUI pair
2. Plot of patient counts over time for target PheCode-CUI pair
3. Plot of intra-patient correlation over time for target PheCode-CUI pair

Notes: (1) Rates are calculated as the number of patients with the target code per calendar year divided by the total number of patients with any code in the same year; (2) Intra-patient correlations are calculated as the Spearman correlation between the code counts for patients in the same year; (3) Black dotted line represent total patient counts per calendar year (denominator of rate).

Module 5: ONCE Feature Trends (Diagnoses, Medications, Labs, Procedures)

This module identifies the top five related codes in different categories (diagnosis, medication, lab, procedure, CUI) based on ONCE feature similarity and tracks their trends over time. Patient counts are summarized annually (line plots) and overall (bar plots).

Outputs

1. Plot of rates over time for related features
2. Plot of patient counts over time for related features